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ABSTRACT 

The  Army  Learning  Model  (ALM)  discusses  the  importance  of  using  valid  and  reliable  assessments  in 
training  technologies.  It  specifically  mentions  the  use  of  pre-tests  to  tailor  training  and  post-tests  to  ensure 
that  learning  has  occurred  to  a  standard.  However,  other  than  these  recommendations,  the  ALM  does  not 
address  how  assessments  should  be  designed,  delivered,  and  otherwise  used  to  maximize  Soldier  training. 
Questions  regarding  which  type  of  assessment  should  be  used,  the  optimal  frequency  of  assessment,  and 
how  to  automate  assessment  in  collaborative  problem-solving  scenarios  remain  to  be  answered.  To  address 
these  issues,  the  U.S.  Army  Research  Institute  (ARI)  and  ICF  Internationa!  developed  prototype  training 
that  provides  a  test-bed  for  conducting  research  on  assessment  strategies  with  maturing  training 
technologies.  The  training  content,  which  teaches  Soldiers  how  to  operate  a  common  piece  of  signal 
equipment,  is  delivered  via  a  mobile  device,  virtual  classroom  and  game-based  training  platform.  The 
mobile  training  focuses  on  declarative  knowledge  and  covers  basic  terminology  and  principles  for 
operating  the  radio;  the  virtual  classroom  covers  more  advanced  procedures  topics;  and  the  collaborative 
scenario  serves  as  a  capstone  exercise  where  Soldiers  appty  the  knowledge  they  gained  from  the  mobile 
and  virtual  classroom  in  a  team-based  problem-solving  task.  Assessments  are  integrated  within  each 
training  platform  to  track  learning.  This  paper  discusses  the  development  of  the  prototype  training  and 
assessments,  including  a  discussion  of  the  prototype  concept,  the  instructional  design  approach  we  used  to 
develop  the  training  and  corresponding  assessments,  and  the  technology  considerations  and  constraints. 
Then,  we  describe  the  resutts  of  a  beta  test  that  examined  the  validity  and  usability  of  the  training  platforms 
and  assessments.  We  conclude  with  a  discussion  of  future  research,  which  examines  critical  questions 
regarding  the  design  and  delivery  of  assessments  within  the  prototype  training. 
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INTRODUCTION 

The  Army  Learning  Model  (ALM)  sets  forward  an  agenda  for  innovation  in  Army  training  where  instructor- 
centered  training  is  replaced  by  learner-centered  training  (TRADOC,  201 1).  Rather  than  limiting  training  to  specific 
timeframes  and  locations,  the  ALM  calls  for  a  training  system  that  can  be  accessed  at  the  ‘point  of  need’  and  one, 
“...that  extends  knowledge  to  Soldiers  at  the  operational  edge,  is  capable  of  updating  learning  content  rapidly,  and  is 
responsive  to  Operational  Army  needs”  (TRADOC,  201 1,  p,  16).  Furthermore,  it  calls  for  training  to  be  tailored  to 
the  needs  of  the  learner  in  order  to  maximize  training  efficiency  and  effectiveness.  A  key  element  of  this  vision  is 
the  use  of  distributed  learning  technologies  including  mobile,  virtual,  and  game-based  training  platforms  to  support 
learning  at  the  point  of  need.  Another  critical  piece  is  the  integration  of  assessments  within  these  technologies  to 
ensure  learning  has  occurred  to  a  standard  and  training  is  tailored  to  the  learner’s  level  of  experience  and 
competence. 

Technologies  that  facilitate  the  type  of  learner-centered  training  advocated  by  the  ALM  are  advancing  quickly.  For 
example,  researchers  cite  a  number  of  potential  advantages  of  using  mobile  learning  technologies  for  training  and 
education,  such  as  giving  learners  more  flexible  access  to  learning  materials,  which  can  increase  learning  gains  (c.f., 
Holden  &  Sykes,  2011;  Norris  &  Soloway,  2004;  Rochelle  &  Pea,  2002),  and  providing  a  means  for  assessing 
performance.  However,  researchers  are  only  beginning  to  investigate  optimal  use  and  effectiveness  of  these  types  of 
technologies  for  these  purposes. 

The  educational  and  training  literature  also  discusses  the  benefits  of  leveraging  simulated  and  virtual  learning 
environments  (SLE/VLE)  and  videogames  for  training.  Specifically,  research  shows  that  VLEs  and  games  can  lead 
to  increases  in  trainee  motivation  and  engagement,  thus  increasing  the  desire  to  learn,  the  desire  to  practice,  and 
other  training  related  outcomes  (Mautone,  Spiker,  Karp,  &  Conkey,  2010;  Topolski,  Leibrecht,  Cooley,  Rossi, 
Lampton,  &  Knerr,  2010).  Further  research  shows  that  embedding  and  delivering  assessments  within  these 
environments  is  critical  for  learning  (Bedwell  &  Salas,  2010;  Johnson  &  Mayer,  2009).  Still,  questions  regarding 
what  type  of  assessment  should  be  used,  the  frequency  of  assessment  delivery,  and  how  to  best  automate  assessment 
in  VLEs  for  collaborative  problem-solving  scenarios  remain  to  be  answered.  Research  into  these  areas  is  required  if 
the  Army’s  vision  for  a  Soldier-centered  learning  environment  is  to  succeed. 

In  response  to  these  needs,  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Science  (ARI),  in 
collaboration  with  the  U.S.  Army  Research  Laboratory’s  Simulation  and  Training  Technology  Center  (ARL-STTC) 
and  ICF  International,  have  developed  prototype  training  that  provides  a  test-bed  for  conducting  research  on 
assessment  strategies  in  maturing  learning  technologies,  including  mobile  devices,  virtual  classrooms,  and 
SLE/VLEs.  This  paper  provides  a  case  study  documenting  the  development  of  the  training  and  assessment 
prototypes.  We  include  a  discussion  of  the  prototype  concept,  the  instructional  design  approach  used  to  develop  the 
training  and  corresponding  assessments,  the  technology  considerations  and  constraints,  and  the  assessment  strategy 
used  within  each  training  platform.  We  also  discuss  the  results  of  a  beta  test  that  captured  Soldiers’  reactions  to 
using  the  prototypes  in  an  institutional  training  setting.  The  paper  concludes  with  a  discussion  of  our  current  line  of 
research. 

PROTOTYPE  CONCEPT 
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The  content  selected  for  this  project  involved  the  assembly,  configuration,  programming,  and  operation  of  a  widely 
used  combat  radio,  the  AN/PRC-148  Joint  Enhanced  Multiband  Inter/Intra  Team  (JEM)  radio.  Currently,  training  on 
the  JEM  is  conducted  over  the  course  of  one  day  at  the  Signal  Regimental  Non-Commissioned  Officer  (NCO) 
Academy  at  Ft.  Gordon,  Georgia.  During  the  one-day  course,  a  class  of  approximately  30  NCOs  participates  in 
PowerPoint-based  didactic  instruction,  followed  by  hands-on  practice  with  the  radio.  At  the  conclusion,  students 
take  an  open-book  test  to  determine  whether  they  pass  or  fail  the  course.  The  goal  of  this  project  was  to  transition 
this  training  content  to  a  VLE,  leveraging  mobile,  virtual  classroom,  and  collaborative  technology,  and  to  integrate 
learning  assessments  throughout  the  training. 

In  transforming  the  JEM  training  from  classroom-based  instruction  to  mobile,  virtual,  and  collaborative  training 
(and  the  corresponding  suite  of  assessments),  the  team  used  an  integrated  approach.  Our  concept  for  the  training  and 
assessments  was  for  the  training  presented  in  each  platform  to  build  upon  one  another  and  for  the  assessments  to 
track  learning  within  and  between  modules.  Specifically,  a  computer  adaptive  test  (CAT)  is  used  to  track  learning 
progress  between  the  mobile  and  the  virtual  classroom  training.  Interim  assessments,  referred  to  as  check  on 
learning  activities  are  used  to  measure  learning  within  each  platform.  A  collaborative,  capstone-like  exercise  is  used 
to  measure  learning  and  performance  at  the  end  of  training  at  both  the  individual  and  team  levels.  All  components 
are  coordinated  and  aligned  using  an  overarching  instructional  systems  design  (ISD)  map  and  a  comprehensive 
technology  architecture  plan.  All  portions  of  the  training  are  integrated  into  the  Soldier-Centered  Army  Learning 
Environment  (SCALE),  a  prototype  data-driven  training  architecture  and  learning  management  system  (Mangold, 
Beauchat,  Long,  &  Amburn,  2012).  Figure  1  provides  an  overview  of  our  concept. 
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Figure  1.  Prototype  concept. 


Critical  to  the  success  of  this  effort  were  an  adherence  to  sound  instructional  design,  integration  of  employed 
technologies,  and  use  of  an  assessment  strategy  that  mapped  closely  to  the  learning  objectives  and  to  the  goals  of  the 
ALM.  Our  approach  to  these  three  dimensions  is  described  in  the  sections  that  follow. 


INSTRUCTIONAL  DESIGN  APPROACH 

SLE/VLEs  can  only  reflect  ISD  principles  if  they  are  designed  using  those  principles.  A  critical  component  of  this 
effort  was  that  all  assessment  components  and  ail  training  components  were  carefully  planned  and  aligned.  One  of 
the  first  tasks  was  to  identify  (and  in  some  cases  modify)  the  learning  objectives  related  to  the  assembly, 
configuration,  programming,  and  operation  of  the  JEM.  The  learning  objectives  were  decomposed  into  terminal, 
subordinate,  and  enabling  objectives  that  were  visually  depicted  on  an  ISD  map. 
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The  ISD  map  was  used  as  a  blueprint  for  many  segments  of  the  training  development  process,  as  it  visually  depicted 
the  concepts  that  were  to  be  trained,  the  sequencing  of  content,  and  the  training  delivery  platform  for  each  objective. 
The  map  was  also  used  to  identify  missing  learning  objectives  and  to  determine  which  objectives  were  out  of  scope 
for  our  specified  audience.  In  addition,  it  illustrated  which  aspects  of  the  training  program  covered  declarative, 
cognitive,  psychomotor,  or  procedural  objectives. 

In  the  case  of  the  JEM  training,  the  ISD  map  revealed  three  terminal  learning  objectives  that  progressed  from 
learning  basic  features  and  components  of  the  radio,  to  demonstrating  how  to  operate  the  radio,  and  finally  to 
troubleshooting  the  radio.  Each  terminal  objective  was  decomposed  further  into  subordinate  objectives.  In  addition, 
the  map  was  used  to  align  assessments  with  training  content.  Specifically,  we  identified  four  primary  content 
domains  that  spanned  across  the  learning  objectives  (e.g.,  Terminology  and  Specifications,  Functionality/Basic 
Procedures,  Complex  Procedures,  and  Troubleshooting).  These  will  be  described  in  the  assessment  approach. 

The  ISD  map  also  enabled  the  team  to  determine  which  learning  objectives  were  best  taught  in  an  informal  manner, 
using  a  self-directed  mobile  platform,  in  a  more  guided  manner,  using  a  virtual  classroom  platform,  and  in  a 
collaborative  manner,  using  a  multi-player  virtual  environment.  The  learning  objectives  that  were  declarative  in 
nature  or  moderately  demonstrative  (i.e.,  Identify  the  battery  latch  and  explain  when  it  is  used  and  how  to  use  it) 
were  determined  to  be  more  conducive  to  the  mobile  platform  as  this  medium  can  be  used  to  present  didactic  content 
and  interactive  assessments  (i.e.,  multiple  choice  quizzes,  interactive  radio  tasks).  The  objectives  that  involved 
programming  the  radio  were  determined  to  be  conducive  to  the  virtual  classroom,  as  an  instructor  would  be  able  to 
demonstrate  how  to  manipulate  the  radio  for  more  complex  concepts.  Finally,  those  objectives  which  were  more 
evaluative  in  nature  and  required  critical  thinking  were  aligned  best  with  the  collaborative  scenario. 


TRAINING  TECHNOLOGY  DEVELOPMENT  APPROACH 

While  the  ISD  map  was  being  created  to  serve  as  a  blueprint  for  the  content,  we  also  developed  a  parallel  technology 
architecture  plan  to  support  the  integration  of  the  mobile,  virtual  classroom,  and  collaborative  training  and 
assessment  technologies.  A  goal  of  our  development  approach  was  to  create  consistency  between  all  of  the 
prototype  components  while  capitalizing  on  the  unique  features  each  training  and  assessment  mode  provided.  This 
was  important  for  creating  a  cohesive  user-experience,  and  for  developing  valid  assessments  and  effective  training. 
Figure  2  provides  a  high-level  outline  of  our  technology  architecture. 
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Figure  2.  High  Level  Architecture 


The  first  step  in  our  technology  development  approach  was  to  determine  which  technologies  would  best  support  the 
training  and  assessment  methodology.  In  order  to  generate  a  cohesive  and  uniform  approach  to  development,  we 
evaluated  technologies  that  adhered  to  the  following  criteria: 

•  Use  a  'Build  Once  "  approach  to  development :  The  technology  framework  supported  multiple  platforms 
including  mobile,  web,  and  personal  computer 

•  Choose  thin  client  systems  with  efficient  load  and  run  times 

•  Minimize  language  integration :  The  number  of  computer  programming  languages  used  in  this  development 
was  be  held  to  a  minimum  to  simplify  the  development  process 

•  Use  DoD  accredited  technologies  and  standards  to  the  extent  possible-.  In  order  to  build  towards  a  system 
that  can  rapidly  transition  to  operational  use,  the  technology  must  generate  approved  Certificate  of 
Networthiness  (CoN)  content. 
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Based  on  the  above  criteria,  a  training  framework  was  designed  and  implemented  as  three  separate,  yet  integrated, 
applications  -  a  mobile  iOS  application,  a  virtual  classroom,  and  a  collaborative  multiplayer  virtual  world.  The 
framework  was  designed  to  share  content,  assessments,  and  scores  across  the  three  platforms.  For  example,  a 
learner  may  embark  on  mobile  training  as  a  prerequisite  and  scores  on  the  mobile  training  post-test  would  inform  the 
item  selection  for  the  virtual  classroom  post-test.  Assessment  scores  from  each  platform  were  stored  in  the  SCALE 
prototype  learning  management  system,  which  provided  a  method  for  tracking  and  evaluating  student  learning  skills. 

Mobile  Application 

The  mobile  application  (see  Figure  3)  was  designed  to  teach  introductory 
facts  and  concepts  regarding  the  radio  through  a  combination  of  didactic 
learning  content  and  simple  hands-on  use  of  a  virtual  radio.  Although  the 
training  was  designed  to  run  on  iOS  devices,  specifically  the  iPad,  the  core 
content  and  technologies  are  portable  to  other  tablet  devices,  web-based  and 
stand-alone  PC  and  Mac  environments.  A  key  feature  in  the  application  is  a 
functional  3D  model  of  the  JEM,  This  model  allows  the  user  to  practice 
functions  with  a  simulated  radio.  For  example,  a  learner  using  the  iPad 
application  is  able  to  program  the  radio  by  actually  pressing  the  buttons  on 
the  virtual  radio.  The  model  was  leveraged  throughout  all  modes  of  training  Figure  3.  Mobile  Application 
and  the  assessments. 


Virtual  Classroom  Application 


The  virtual  classroom  technology  extends  aspects  of  the  core  mobile 
application  to  include  the  ability  to  share  and  deliver  content  in  a  ‘virtual 
schoolhouse’  style  interface  (see  Figure  4).  An  instructor  can  lead  the  virtual 
classroom  course  using  text  chat,  voice  over  IP,  and  has  the  ability  to  both 
manipulate  a  virtual  radio  and  share  content  with  students  in  the  classroom. 
Students  can  communicate  with  the  instructor,  interact  with  course  material, 
and  complete  assessments  throughout  the  course.  Once  again,  all 
performance  data  is  stored  in  the  SCALE  architecture  and  the  3D  model  of  the 
radio  was  leveraged  throughout. 


Figure  4.  Virtual  Classroom 


Collaborative  Application 


The  collaborative  scenario  was  developed  so  that  students  can  apply  the 
information  learned  from  the  mobile  application  and  virtual  classroom.  The 
collaborative  application  extends  features  of  the  mobile  application  and  virtual 
classroom  to  include  realistic  scenario-driven  vignettes  where  students  work 
in  teams  to  collaboratively  resolve  issues  related  to  manipulating  and 
troubleshooting  the  JEM  radio  (see  Figure  5).  The  training  and  assessment 
technologies  utilized  in  the  virtual  classroom  and  mobile  app  were  also 
leveraged  in  the  collaborative  technology  framework,  This  facilitated 
consistent  user-interface  experience,  reduced  cognitive  load,  and  minimized 
development  cost. 


Figure  5.  Collaborative  Scenario 


ASSESSMENT  APPROACH 

A  primary  purpose  of  this  effort  was  to  develop  a  suite  of  assessments  that  could  measure  learning  progress,  provide 
feedback  in  order  to  support  and  reinforce  learning,  be  integrated  into  the  training  technologies,  and  be  utilized  for 
future  research.  The  suite  of  assessments  developed  included  a  CAT  administered  at  three  points  throughout  the 
training  and  interim  assessments  (called  “checks  on  learning,”  adopting  a  term  frequently  used  in  Army  training) 
that  were  administered  periodically  within  each  training  platform.  It  also  included  a  collaborative-scenario  that 
served  as  a  capstone  assessment  exercise.  In  response  to  the  vision  set  out  by  ALM,  the  CAT  was  designed  such 
that  the  test  could  eventually  be  used  to  allow  learners  to  test-out  of  portions  of  the  training,  while  the  interim 
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assessments  and  the  scenario-based  collaborative  exercise  were  designed  such  that  instructors  would  be  able  to 
ensure  learning  occurred  to  a  standard.  The  three  types  of  assessments,  and  our  approach  to  developing  each,  are 
described  in  detail  below. 

Computer  Adaptive  Testing 

To  complement  the  mobile  device  training  and  the  virtual  classroom  modules,  we  developed  a  CAT  to  be 
implemented  at  different  time  points  for  evaluating  learning  progress.  Specifically,  the  CAT  was  designed  to  serve 
as  the  pre-  and  post-tests  at  the  beginning  of  the  mobile  training,  upon  completion  of  the  mobile  training,  as  well  as 
after  the  virtual  classroom  training. 

We  employed  item  response  theory  (IRT)  and  adaptive  testing  principles  as  the  base  psychometric  technology  for 
the  CAT  (Harvey  &  Hammer,  1999).  One  advantage  of  IRT  is  that  scoring  does  not  require  trainees  to  receive  the 
same  sequence  of  items;  instead  a  subset  of  items  is  chosen  from  an  item  pool  to  gauge  levels  of  competency. 
Consequently,  trainees  can  take  an  assessment  more  than  once  without  seeing  the  same  exact  items.  In  addition,  a 
CAT  maximizes  testing  efficiency  by  identifying  the  most  diagnostic  items  for  each  trainee  using  adaptive  item 
selection.  This  technique  can  reduce  test  length  by  approximately  50%  without  losses  in  measurement  precision 
(Weiss  &  Kingsbury,  1984), 

Development  of  the  computer  adaptive  test.  The  CAT  item  pool  was  designed  to  vary  sufficiently  in  difficulty  to 
accurately  assess  individuals  across  all  proficiency  levels  in  every  content  domain  relevant  to  the  JEM  training.  To 
achieve  this  goal,  we  first  reviewed  the  JEM  1SD  Map  and,  based  on  the  subordinate  learning  objectives, 
synthesized  the  content  into  four  main  content  domains:  Terminology  and  Specifications,  Functionality/Basic 
Procedures,  Complex  Procedures,  and  Troubleshooting.  A  test  plan  for  each  domain  was  developed  based  on  the 
enabling  learning  objectives  identified  in  the  JEM  1SD  Map.  This  plan  was  used  to  guide  the  development  of  test 
items.  For  each  domain,  we  developed  a  subset  of  draft  items  that  varied  in  difficulty  (easy,  medium  and  difficult). 
Then,  instructors  at  Ft.  Gordon  rated  the  relative  level  of  difficulty  of  each  item  on  a  7-point  scale  from  very-low  to 
very-high;  these  ratings  served  as  the  item-level  parameters  in  the  CAT.  Items  were  further  validated  in  a  subject 
matter  expert  (SME)  review  workshop,  finalized,  and  then  loaded  onto  the  CAT  technical  platform  for 
implementation  into  the  training. 

Checks  on  Learning 

In  addition  to  the  CAT,  we  developed  a  series  of  interim 
assessments  (check  on  learning  activities)  to  track  learning 
within  the  mobile  and  virtual  classroom  applications.  The 
purpose  of  the  check  on  learning  activities  was  to  measure 
understanding  of  the  previously  presented  training  material, 
provide  an  opportunity  to  practice  applying  the  knowledge, 
and  provide  feedback  as  the  learner  completed  the  training. 

The  check  on  learning  items  were  developed  with  the  same 
technology  used  in  the  mobile  and  virtual  classroom 
trainings,  including  a  3D  model  of  the  JEM.  One  goal  in 
creating  these  assessments  was  to  leverage  simulation 
technology  to  create  higher-fidelity  assessment  items  than 
more  traditional  formats  (like  multiple-choice)  often  afford. 

For  example,  the  assessment  item  displayed  in  Figure  6 
requires  students  to  program  a  virtual  JEM  radio.  Students 
interact  with  the  virtual  radio  in  much  the  same  way  they  do 
a  real  radio;  they  press  and  hold  buttons  and  manipulate  the 
radio’s  attachments  and  positioning.  The  technology 
assesses  student  performance  and  provides  feedback  an 
incorrect  answer  is  submitted. 

Development  of  mobile  checks  on  learning.  For  each  enabling  learning  objective  in  the  mobile  training,  we  created 
a  bank  of  check  on  learning  assessments.  These  assessments  were  administered  at  roughly  equal  intervals 
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Figure  6.  Example  Check  on  Learning  Item 
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throughout  the  mobile  and  training  and  varied  in  length  from  2  to  1 1  items.  Each  check  on  learning  assessment  only 
focused  on  content  covered  in  the  training  that  immediately  preceded  that  assessment,  and  learners  were  not  allowed 
to  go  back  to  previous  training  content  after  beginning  a  check  on  learning  assessment.  The  assessment  items  were 
developed  in  seven  different  formats.  These  formats  were  designed  to  provide  learners  with  a  variety  of  item  types, 
force  learners  to  think  about  the  content  in  different  ways,  practice  applying  their  knowledge  in  varied  and  unique 
ways,  and  utilize  the  available  technology.  Table  1  provides  a  listing  of  the  different  item  formats  and  a  description 
of  each. 


Table  1.  Check  on  Learning  Item  Formats  and  Descriptions 


Item  Format 

Description 

Multiple  Choice 

Learner  selects  the  one  best  response  from  a  list  of  options.  Some 
items  include  pictures  in  the  item  stem  or  response  options. 

Choose  all  that  apply  -  Multiple  Choice 

Fonnatted  like  a  multiple  choice  item,  except  the  learner  can  select  as 
many  options  as  desired.  Some  items  include  images. 

Interactive  Multiple  Choice 

Item  includes  a  bar  with  arrows  or  a  movable  icon  that  the  learner  can 
slide  to  indicate  the  answer. 

Matching 

Two  or  more  items  are  presented  together  with  multiple  response 
options.  Each  item  has  only  one  correct  answer.  When  an  option  is 
dragged,  it  should  disappear  from  the  option  list.  Example  items  may 
include  a  virtual  JEM  that  require  the  learner  to  click  on  the  JEM  to 
identify  specified  components,  to  drag  labels  to  the  appropriate 
location  on  the  JEM,  or  to  connect  external  components  to  the  JEM 

Choose  all  that  apply  -  Matching 

Two  or  more  items  are  presented  together.  Response  options  are 
displayed  underneath.  Learner  can  drag  option(s)  up  to  applicable 
item.  When  an  option  is  dragged,  the  option  remains  below  so  it  could 
be  used  again  for  the  other  item.  Some  items  include  images. 

True  or  False 

Learner  clicks  on  True'  or  'False'  in  response  to  the  item. 

Virtual  JEM  Programming 

Items  include  a  virtual  JEM  that  the  learner  must  program  using  the 
radio’s  keypad. 

Most  items  are  worth  one  point  each;  however,  several  items  included  multiple  parts  and  are  therefore  worth 
multiple  points.  Following  each  item,  the  learner  receives  feedback  that  provides  useful  information  about  the 
correct  answer.  The  feedback,  a  key  feature  of  assessment  approach,  reinforces  learning.  At  the  end  of  a  check  on 
learning  assessment,  the  learner  can  review  a  summary  of  the  items  they  answered  correctly  and  incorrectly. 

Development  of  virtual  classroom  checks  on  learning.  The  check  on  learning  activities  were  similar  for  both  the 
mobile  and  virtual  classroom  applications.  The  key  difference  between  the  two  resulted  from  the  difference  in  the 
terminal  objectives  between  the  mobile  training  and  virtual  classroom  training.  Specifically,  the  mobile  training 
included  a  stronger  focus  on  the  recall  of  declarative  knowledge  (i.e.,  facts,  rules,  concepts),  whereas  the  virtual 
classroom  required  learners  to  apply  that  knowledge.  Therefore,  the  virtual  classroom  included  a  greater  ratio  of 
items  that  incorporated  the  Virtual  JEM  Programming  format.  These  items  simulated  hands-on  practice  with  the 
JEM.  The  ability  to  demonstrate  learning  on  a  simulated  JEM  was  a  key  feature  of  our  assessment  approach. 

Theoretical  rationale  for  aligning  assessment  types  with  training  platforms.  The  decision  to  use  more  multiple 
choice,  matching,  and  true  false  items  in  the  mobile  training  and  more  interactive  assessments  in  the  virtual 
classroom  was  based  the  theoretical  alignment  between  the  learning  objectives  and  learning  outcomes  for  each 
training  platform.  As  mentioned  previously,  there  is  a  stronger  focus  on  declarative  knowledge  in  the  mobile 
training.  Declarative  knowledge  forms  the  foundation  for  more  advanced  knowledge  such  as  procedural  knowledge 
and  critical  thinking  (Anderson,  1982).  Because  of  this,  measuring  the  retention  of  declarative  knowledge  is  most 
appropriate  in  the  initial  stages  of  training  (Kraiger,  Ford  &  Salas,  1993).  An  additional  benefit  of  measuring 
declarative  knowledge  in  the  early  stages  of  training  is  that  declarative  knowledge  assessments  help  to  identify 
knowledge  gaps  that  may  hinder  individuals  from  formulating  higher-order  knowledge. 
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As  knowledge  advances  beyond  declarative  knowledge,  several  interrelated  changes  in  processing  occur.  First, 
learners  begin  to  focus  less  on  rules  and  facts  and  more  on  the  nuances  of  applying  that  knowledge  (Anderson, 
1982).  Second,  individuals  begin  to  develop  more  meaningful  schemas  that  describe  the  task.  Such  schemas, 
referred  to  as  mental  models,  allow  individuals  to  describe  the  functions  and  inter-dependences  among  task  elements 
(Kraiger  et  al.,  1993).  Based  on  this  notion,  we  designed  the  virtual  classroom  items  to  assess  how  well  trainees 
could  perform  tasks  on  the  virtual  radio  and  troubleshoot  problems  as  they  occurred. 

Collaborative  Scenario 

The  goal  of  the  collaborative  scenario  is  to  give  learners  an  opportunity  to  apply  what  they  have  learned  in  a  team- 
based  capstone  exercise.  Thus,  this  module  essentially  functions  as  an  assessment.  The  concept  for  this  exercise 
involves  three  players  proceeding  through  a  virtual  “day  in  the  life”  turn-based  scenario  together.  The  exercise 
entails  a  series  of  vignettes  that  focuses  on  the  operation  of  the  JEM  radio  in  real-life  contexts  (e.g.,  preparing  the 
radio  for  mission,  troubleshooting  when  encountering  unusual  circumstances).  Each  vignette  requires  team  members 
to  program  and  troubleshoot  the  radio.  The  radio  control  is  rotated  among  players  throughout  the  exercise  -  for  any 
given  vignette,  only  one  player  is  in  charge  of  the  radio  while  the  other  two  players  watch  the  actions  of  the  player- 
in-charge  in  real  time  (i.e.,  all  players  will  see  the  same  screen  but  only  one  player  will  be  able  to  be  able  to  “use” 
the  radio  while  the  other  two  players  will  be  in  “view  only”  mode).  During  the  action  periods,  there  is  an  open 
communication  channel  (i.e.,  chat  function)  that  allows  for  discussions  among  the  players  as  they  proceed  through 
the  game  so  they  can  make  decisions  collectively. 

There  were  several  advantages  to  this  approach.  First,  the  “day  in  the  life”  approach  offers  an  opportunity  for 
learners  to  apply  what  they  have  learned  from  the  mobile  and  virtual  classroom  trainings,  preparing  them  for 
knowledge  and  skill  transfer  to  the  real  world.  It  also  adds  a  decision-making  dimension  (i.e.,  “when  to  do  what”)  in 
addition  to  the  knowledge  and  skill  dimension  (i.e.,  “how  to”).  This  is  especially  important  for  assessing  critical- 
thinking  skills.  Second,  the  training  technology  was  leveraged  to  provide  a  simulated  operational  environment  that 
is  contextualized  and  high-fidelity.  Third,  the  setup  of  the  exercise  provides  both  structural  and  emergent 
components  of  collaboration  (for  a  comprehensive  discussion  on  formal  structures  versus  emergent  processes,  see 
Kozlowski  and  Klein,  2000).  The  structural  component  (i.e.,  the  radio  being  passed  from  one  player  to  the  next) 
ensures  task  interdependency  because  all  players  need  to  succeed  in  their  scenarios  for  the  team  to  accomplish  the 
overall  goal.  The  emergent  component  (i.e,,  how  the  players  will  work  together  over  the  open  communication 
channel)  reflects  natural  team  processes  and  may  provide  rich  data  to  answer  a  variety  of  research  questions. 

Development  of  collaborative  scenario.  To  develop  the  content  of  the  collaborative  scenario,  we  first  collected 
critical  incidents  (see  Flanagan,  1954)  from  instructors  at  Ft.  Gordon  that  described  in  detail  a  real-to-life  battlefield 
scenario  that  would  involve  using  the  JEM.  Our  goal  was  to  elicit  information  on  the  real-world  contexts  around 
which  certain  JEM  procedures  or  troubleshooting  techniques  covered  in  our  training  would  occur.  In  addition  to 
collecting  these  situational  examples,  we  also  elicited  information  regarding  appropriate  art  assets,  virtual 
environment  components,  player  actions,  and  non-player  character  (NPC)  actions  to  ensure  the  scenarios  included 
an  appropriate  level  of  physical  and  psychological  fidelity.  Based  on  the  information  collected,  we  were  able  to 
confirm  vignettes  within  one  scenario  story  of  a  Personnel  Security  Detachment  (PSD)  mission  where  a  Colonel  is 
visiting  a  local  village  to  check  on  the  progress  of  a  new  school  under  construction.  Various  tasks  related  to  the  use 
of  the  radio  are  required  of  Soldiers  accompanying  the  Colonel. 

Building  on  the  critical  incidents  collected,  we  developed  detailed  vignettes  for  the  collaborative  scenario.  We  used 
several  key  considerations  to  guide  the  vignette  development  process.  First,  we  ensured  that  the  situation  players 
encountered  did  not  result  in  repeated  task  execution.  In  other  words,  the  tasks  required  in  the  vignettes  were  mostly 
independent  to  avoid  over-penalizing  the  same  mistake  (e.g.,  performance  on  one  task  would  not  affect  performance 
on  another  task).  Second,  we  referred  to  the  subordinate  objectives  in  the  JEM  ISD  Map  to  ensure  that  a  variety  of 
tasks  were  covered  throughout  the  vignettes.  Finally,  the  vignettes  were  written  to  not  involve  any  tactical  decision¬ 
making  outside  of  the  scope  of  radio  use  so  that  the  assessments  strictly  adhered  to  the  learning  objectives  and 
training  content. 

In  conjunction  with  the  collaborative  scenario  development,  we  embedded  scoring  into  the  exercise  itself.  Two 
types  of  assessments  were  developed:  individual-level  knowledge  check  questions  and  team-level  radio 
manipulation  tasks.  Individual-level  knowledge  check  items  were  developed  as  single-answer  multiple-choice 
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questions.  Each  player  completes  these  individually  without  communication  with  other  players  (i.e.,  the  chat 
Emotion  would  be  disabled  until  everyone  completes  the  knowledge  check  questions).  Team-level  radio 
manipulation  tasks  were  developed  as  a  series  of  steps  each  player  in  charge  of  the  radio  must  carry  out  in  order  to 
successfully  complete  a  vignette.  For  scenarios  that  involve  troubleshooting  procedures,  players  must  first  identify 
the  problem  and  determine  the  correct  course  of  action,  and  then  perform  the  mechanical  manipulation  steps  to 
correct  the  problem.  The  team  does  not  receive  credit  for  a  vignette  if  the  radio  manipulation  steps  are  performed 
correctly,  but  an  incorrect  course  of  action  is  identified  (e.g.,  if  the  battery  is  malfunctioning,  the  team  will  not 
receive  credit  for  correctly  replacing  the  antenna).  In  other  words,  the  course  of  action  and  the  radio  manipulation 
must  both  be  correct  to  receive  credit  for  the  vignette. 

At  the  end  of  the  development  process,  we  derived  seven  vignettes.  In  each  vignette,  players  are  first  presented  with 
a  video  clip  that  advises  them  about  the  situation  and  radio  setup  for  the  task.  Then  players  are  prompted  to  answer 
several  individual-level  knowledge  check  questions  in  relation  to  the  preceding  video.  Once  all  knowledge  check 
questions  are  completed,  the  player  assigned  as  the  active  player  for  the  vignette  is  granted  control  of  the  simulated 
radio;  the  other  players  watch  the  active  player’s  actions  and  offer  advice  and  input.  Some  radio  manipulation  tasks 
involve  multiple-choice  questions  that  the  active  player  must  answer  correctly  in  order  to  proceed.  For  example, 
when  connecting  or  disconnecting  parts  of  the  radio,  a  question  may  appear  prompting  the  active  player  to  select  the 
correct  physical  motion.  Team  members  may  offer  the  active  player  advice  and  the  active  player  may  make 
repeated  attempts.  The  radio  manipulation  tasks  are  scored  based  on  whether  the  overall  goal  of  the  task  is 
completed  successfully  within  the  time  limit.  If  the  team  is  unsuccessful,  they  are  given  demonstration-based 
feedback  in  the  form  of  animated  videos  of  how  to  correctly  perform  the  task. 

The  finished  vignettes  were  presented  to  a  group  of  SMEs  at  Ft.  Gordon  for  validation  purposes.  SMEs  reviewed 
each  vignette  and  their  corresponding  assessments  and  provided  input  and  feedback  regarding  their  realism  and 
feasibility.  The  final  version  of  the  collaborative  scenario  was  based  upon  this  feedback. 

BETA  TEST 


After  we  developed  full  versions  of  the  mobile  training,  virtual  classroom,  and  collaborative  scenario,  we  conducted 
a  beta  test  with  Soldiers  at  Ft.  Gordon,  GA.  The  beta  test  was  an  opportunity  for  Soldiers  (most  with  prior 
experience  using  the  JEM)  to  review  the  training  and  provide  feedback  on  the  content,  appearance/features, 
functionality,  and  organization  of  the  trainings  and  assessments. 

Participants  and  Method 

A  total  of  21  Soldiers  participated  in  the  beta  test  and  each  Soldier  completed  at  least  one  of  the  three  modalities.  In 
total,  the  mobile  training  was  completed  by  18  Soldiers,  the  virtual  classroom  was  completed  by  eight  Soldiers,  and 
the  collaborative  scenario  was  completed  by  nine  Soldiers.  The  Soldiers  had  an  average  tenure  of  10  years  in  the 
Army,  an  average  of  one  year  in  their  current  rank,  and  had  an  average  of  3.4  deployments.  The  ranks  of  the 
participants  were  split  between  Sergeant  First  Class  (SFC)  (nine  total)  and  Staff  Sergeant  (SSG)  (12  total).  The 
military  occupational  specialties  (MOS)  of  the  participants  varied  between  25U  (12  total),  25C  (seven  total),  25B 
(one  total),  and  25M  (one  total). 

Participants  were  assigned  to  a  training  modality  based  on  the  time  in  w'hich  they  were  available  to  participate.  Only 
two  trainings  could  take  place  at  a  time  (there  were  only  two  rooms)  and  only  three  Soldiers  could  complete  the 
collaborative  scenario  at  a  time.  Participants  were  brought  to  a  room  with  either  iPads  or  desktop  computers.  Those 
with  the  iPads  were  administered  the  mobile  training,  and  those  in  the  room  with  desktops  were  either  administered 
the  virtual  classroom  or  collaborative  scenario. 

After  participants  were  familiarized  with  the  technological  features,  they  proceeded  with  the  training  and 
assessments  and  documented  any  comments  and/or  suggestions  based  on  their  user  experience.  Upon  completion  of 
each  training  modality,  participants  completed  an  anonymous  online  survey  designed  to  measure  their  reactions  to 
the  training.  Specifically,  the  survey  captured  participants’  satisfaction  with  various  dimensions  of  the  prototype 
training,  including  the  overall  enjoyment,  usability  of  technology,  instructional  design,  and  perceived  training  utility 
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and  learning  transfer.  Once  the  survey  was  completed,  participants  were  debriefed  and  final  comments  on  the 
trainings  were  collected. 

Results  of  the  Beta  Test 

Analysis  of  the  learner  reaction  measure  revealed  that  overall,  the  training  was  perceived  positively.  The  majority  of 
participants  indicated  that  they  enjoyed  the  training  and  would  recommend  it  to  others.  With  regard  to  the  ease  of 
use,  accessibility,  functionality,  and  satisfaction  with  the  training  technologies,  over  80%  of  participants  indicated 
favorable  attitudes  towards  the  mobile  and  virtual  classroom  modalities  across  all  questions  on  this  topic.  Attitudes 
were  less  favorable  for  the  collaborative  scenario  modality  with  only  60%  of  participants  endorsing  it.  During 
observations  and  post-training  debriefs  for  the  collaborative  scenario  modality,  we  uncovered  some  usability  issues 
that  were  likely  explanations  for  the  lower  ratings.  These  issues  were  later  addressed  to  improve  the  user-experience. 

In  terms  of  instructional  design,  over  80%  of  participants  endorsed  the  course  design  for  the  mobile  and 
collaborative  scenario  modalities.  Attitudes  were  less  positive  on  the  virtual  classroom  modality  -  only 
approximately  70%  of  participants  indicated  favorability  over  the  course  design  for  this  modality.  During  post¬ 
training  debriefs,  participants  raised  concerns  about  some  of  the  virtual  classroom  content  containing  unnecessary 
details,  especially  on  topics  that  were  basic  and  had  already  been  covered  in  the  mobile  training.  As  a  result  the 
length  of  the  virtual  classroom  modality  was  too  long,  which  could  lead  to  learner  fatigue  and  loss  of  attention.  This 
information  was  used  to  later  shorten  the  virtual  classroom. 

When  asked  about  learning  as  a  result  of  these  training  modalities,  the  majority  of  participants  in  the  mobile 
modality  and  in  the  collaborative  scenario  modality  and  less  than  half  of  participants  in  the  virtual  classroom 
modality  indicated  they  “learned  a  lot.”  In  addition,  with  regard  to  training  utility  and  learning  transfer,  over  half  of 
the  participants  endorsed  the  practical  value  of  the  training  and  its  application  on  the  job  across  all  modalities. 


CONCLUSIONS  AND  FUTURE  RESEARCH 

The  purpose  of  this  effort  was  to  create  a  training  and  assessment  prototype  that  was  consistent  with  the  objectives 
set  forth  by  the  ALM  and  that  leveraged  the  capabilities  of  mobile  and  SLE/VLE  technologies.  Foundational  to  our 
approach  were  an  adherence  to  sound  instructional  design  principles,  integration  of  employed  technologies,  and 
alignment  of  assessment  strategies  with  learning  objectives  and  technological  affordances.  After  creating  an  ISD 
map  based  on  the  current  training  materials  and  SME  input,  our  project  team  matched  training  objectives  to  training 
platforms  (i.e.,  mobile,  virtual  classroom,  and  virtual  environment),  based  on  the  alignment  of  the  objectives  and  the 
affordances  of  the  delivery  method.  Our  project  team  then  developed  the  training  environments,  training  content, 
and  assessments  in  close  coordination  with  each  other  and  with  SMEs  at  the  Signal  NCO  Academy  at  Ft.  Gordon. 
The  prototype  trainings  built  upon  one  another  and  included  a  mobile  application,  a  virtual  classroom,  and  a 
collaborative  scenario-based  exercise  that  served  as  a  capstone  event.  The  suite  of  assessments  included  a  CAT 
administered  at  three  points  over  the  course  of  training  and  interim  assessments  that  were  administered  periodically 
throughout  the  training.  It  also  included  both  individual  and  team-level  scoring  embedded  in  the  collaborative 
scenario.  The  training  modules  and  assessments  were  designed  and  implemented  as  separate,  yet  integrated, 
applications.  The  overarching  framework  was  designed  to  share  content,  assessments,  and  scores  across  the  three 
platforms.  Finally,  we  tested  our  prototype  with  a  sample  of  Army  Soldiers  in  an  institutional  setting  to  assess 
reactions  to  the  prototype  training.  Generally  speaking,  the  results  demonstrated  that  the  Soldiers  expressed  a 
positive  view  of  the  training  technology,  instructional  design,  and  overall  training  exercise. 

The  prototypes  developed  for  this  effort  are  currently  being  used  to  answer  several  interrelated  questions  related  to 
the  efficacy  of  different  assessment  strategies.  This  line  of  research  specifically  seeks  to  determine  if: 

•  The  prototype  training  is  at  least  as  effective  as  the  traditional  classroom  method. 

•  CATs  provide  value  over  non-adaptive  assessments  in  an  Army  training  contents 

•  Embedding  check  on  learning  activities  into  mobile  and  virtual  classroom  training  effect  learning  and 
retention. 

•  Performance  in  the  simulation  translate  to  better  performance  with  a  real  radio 
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•  There  any  negative  effects  of  practicing  on  a  simulated  radio. 


Answers  to  these  questions  will  inform  implementation  of  the  ALM  and  will  increase  our  understanding  of  the  most 
appropriate  ways  to  use  existing  technologies  for  training  and  assessment.  As  the  Army  continues  to  transform  their 
training  approach,  these  and  similar  questions  will  need  to  be  answered  in  order  to  create  effective  Soldier-centered 
training. 
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